Annotating Disengagement for Spoken Dialogue Computer Tutoring

Within tutoring systems research, there has been a lot of recent interest in developing systems that adapt their responses to the student’s changing affect and attitude as conveyed during the human-computer interaction (e.g., (Forbes-Riley & Litman, 2010b; Conati & Maclaren, 2009; D’Mello et al., 2008; McQuiggan et al., 2008b; Porayska-Pomsta et al., 2008; Wang et al., 2008; Arroyo et al., 2007; Pon-Barry et al., 2006; Gratch & Marsella, 2003; de Vicente & Pain, 2002; Kort et al., 2001)). The hypothesis underlying this research is that responding to student affect and attitude will improve system performance, particularly as measured by student learning. However, this is a challenging task, which usually involves three main steps. The first step involves identifying the target affect/attitude state and labeling it in a dataset of student-system interactions. Typically, these target states are not among the “six basic emotions” (i.e., anger, disgust, fear, happiness, sadness, surprise) (Ekman & Friesen, 1978) that have received significant attention in the wider psychological literature on emotion. Tutoring researchers have shown via annotation studies of interactions between students and tutoring systems that a different range of affect and attitude is displayed by tutoring system users (e.g., (Lehman et al., 2008)). States that have been reported as relevant to tutoring systems are numerous and overlapping and include uncertainty, confusion, self-efficacy, irritation, frustration, boredom, disengagement, curiosity, flow, and interest, among others. Tutoring researchers have further shown that some states, such as uncertainty, confusion, and boredom, correlate with learning and thus are of particular interest from a performance point of view (Craig et al., 2004; Forbes-Riley et al., 2008). However, the best way to label students’ internal affective state(s) is still an open question. Many learning systems researchers rely on trained judges (e.g., (Pon-Barry et al., 2006; Porayska-Pomsta et al., 2008)) while others use student self-reports